SD6337/S92307 
October 24, 2000 
Express Mail No.: EL027381824US 

o 

ASSISTANT COMMISSIONER FOR PATENTS ^ 
BOX PATENT APPLICATION 

WASHINGTON, D.C. 20231 ^jo 

O 

UTILITY PATENT APPLICATION TRANSMITTAL 
Under 37 CFR 1-53(b) 



Transmitted herewith for filing is the patent application of 

INVENTOR(S): BARRY L. SPLETZER. GEORGE 3. DAVIDSON, DANIEL J. 

ZIMMERER, and LISA C. MARRON 

FOR: FRACTIONAL SCREEN VIDEO ENHANCEMENT APPARATUS 

APPLICATION ELEMENTS 

1. ^ Fee Transmittal Form (original + duplicate) 

2. lEI Specification 

Total Pages 32 

3. 13 Drawings 

Total Sheets 02 

4. S Oath or Declaration (with Power of Attorney) Total Pages _3 

a. ^ Newly Executed 

b. □ Copy from Prior Application 

(Continuation or Divisional) 

□ Deletion of Inventors 

Signed Statement Attached Deleting Inventor(s) named in 
prior Application. 37 CFR 1.63(d)(2) and 1.33(b). 

5. □ Incorporation by Reference (if Box 4b is checked). The entire 

disclosure of the prior application(s) from which a copy of the 
Oath or Declaration is supplied under Box 4b, is considered as 
being part of the disclosure of the accompanying application and 
Is hereby incorporated by reference therein. 

6. □ Microfiche Computer Program (Appendix) 



1 



II. ACCOMPANYING APPLICATION PARTS 

7. □ Assignment Papers (cover sheet and document(s) 

8. □ 37 CFR 3.73(b) Statement Kl Power of Attorney 

(where there is an assignee) (with the Declaration) 

9. ^ Information Disclosure Statement lEl Copies of IDS 

(1DS)/PT0-1449 Citations 

10. □ Preliminary Amendment 

11. [El Return Receipt Postcard (Itemized) 

12. □ Other 

III. IF A CONTINUING APPLICATION 

□ Continuation □ Divisional □ Continuation-ln-Part (CIP) 
of prior application No. / 

IV. CORRESPONDENCE ADDRESS 

Timothy D. Stanley 

Sandia National Laboratories 

P. O. Box 5800 - MS-0161 

Albuquerque, New Mexico 87185-0161 

Phone: (505) 844-8600 
FAX: (505) 844-1418 



Dated: 




Gerald Grafe, Reg. No. 42.599 
Sandia National Laboratories 
Patent & Licensing Center. 11500 
P. O. Box 5800 / MS 0161 
Albuquerque. NM 87185-0161 
Telephone: (505) 284-4404 



CERTIFICATION UNDER 37 CFR 1.10 

I hereby certify that this New Application Transmittal and the documents referred to as enclosed therein are being 
deposited with the U. S. Postal Service on the 24th day of October, 2000, in an envelope as "Express Mail Post Office to 
Addressee" Mailing Label Number EL027381824US, addressed to the: Assistant Commissioner for Patents, Box Patent 



Application, Washington, D.G. 20231. 




Marilynn A. Gabe 



2 



PTO/SB/17 (12-98) 
Approved for use through 09/30/2000 0MB 0651-0032 
Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE 
Under the Papenvork Reduction Act of 1995, no persons are required to respond to a coiiection of information unless it displays a valid OMB control number. 



FEE TRANSMITTAL 
FOR FY 2001 

Patent fees are subject to annual revision. 
Small Entity payments must be supported by a small entity statement, 
Otherwise large entity fees must be paid See Forms PTO/SB/09-12. 



Complete if Known 



Application Number 



Filing Date 



First Named Inventor 



Examiner Name 



Group /Art Unit 



10/24/00 



Spletzer, et al. 



TOTAL AMOUNT OF PAYMENT 



{%)710.00 



Attorney Docket No. 



SD6337/S92307 



METHOD OF PAYMENT (check one) 



The Commissioner is hereby authorized to charge 

1 . ^ indicated fees and credit any over pa yments to: 

Deposit Account 
Number 

Deposit Account 
Name 



19-0131 



Sandia Corporation 



□ Charge Any Additional 
Fee Required Under 
37 CFR 1.16 and 1.17 

2. Payment Enclosed: 

□ □ Check □ Money Order □ Other 



FEE CALCULATION 



1. BASIC FILING FEE 

Large Entity Small Entity 
Fee Fee Fee Fee 
Code ($) Code ($) 


Fee Description 


Fee 
Paid 


101 


710 


201 


355 


Utility filing fee 


710 


106 


320 


206 


160 


Design filing fee 




107 


490 


207 


245 


Plant filing fee 




108 


710 


208 


355 


Reissue filing fee 




114 


150 


214 


75 


Provisional filing fee 





SUBTOTAL (1) $710 



FEE CALCULATION (continued) 

3. ADDITIONAL FEES 

Large Entity Smai Entity 
Fee Fee I Fee 



Code ($) 



Fee 
Cod 



($) 



Fee Description 



Fee 
Paid 



2. EXTRA CLAIM FEES 

Extra Claims 

-20**= @ X § 



Fee from 
below 



Total Claims 112| 

Independent § 
Claims 

Multiple Dependent 



- 3**= @ X g 



Fee Paid 

= 0 



**ornumber previously paid, if greater, For Reissues, see below. 
Large Entity Small Entity 

Fee Description 

Claims in excess of 20 
independent d aims in excess of 3 
Multiple dependent claim, if not paid 

**Reissue independent claims 
over ongmal patent 

**Reissue claims in excess of 20 
and over ongmal patent 



Fee 


Fee 


Fee 


Fee 


Code 


($) 


Code 


($) 


103 


18 


203 


9 


102 


80 


202 


40 


104 


270 


204 


135 


109 


80 


209 


40 


110 


18 


210 


9 



SUBTOTAL (2) {$) 



105 
127 


130 
50 


e 

205 
227 


65 
25 


Surcharge - late filing fee or oath 
Surcharge - late provisional filing fee or cover 
sheet 


139 


130 


139 


130 


Non-English specification 


147 
112 

1 1 o 


2,520 
920* 


147 
112 
11 


2,520 For filing a request for reexamination 

g20* R6Q*J®s*'"9 publication of SIR prior to 

Examiner action 
1 ,840 Requesting publication of SIR after Examiner 


115 


110 


215 


55 


Extension for reply within first month 


116 


390 


216 


195 


Extension for reply within second month 


117 


890 


217 


445 


Extension for reply within third month 


118 


1,390 


218 


695 


Extension for reply within fourth month 


128 


1,890 


228 


945 


Extension for reply within fifth month 


119 


310 


219 


155 


Notice of Appeal 


120 


310 


220 


155 


Filing a brief in support of an appeal 


121 


270 


221 


135 


Request for oral hearing 


138 


1,510 


138 


1,510 Petition to institute a public use proceeding 


140 


110 


240 


55 


Petition to revive - unavoidable 


141 


1,240 


241 


620 


Petition to revive - unintentional 


142 


1,240 


242 


620 


Utility issue fee (or reissue) 


143 


440 


243 


220 


Design issue fee 


144 


600 


244 


300 


Plant issue fee 


122 


130 


122 


130 


Petitions to the Commissioner 


123 


50 


123 


50 


Petitions related to provisional applications 


126 


240 


126 


240 


Submission of Information Disclosure Stmt 


581 


40 


581 


40 


Recording each patent assignment per property 
{times number of properties) 


146 


710 


246 


355 


Filing a submission after final refection 
(37 CFR 1.129(a)) 


149 


710 


249 


355 


For each additional Invention to be examined 
(37 CFR 1.129(b)) 



Other fee (specify) 
Other fee (specify) 



*Reduced by Basic Filing Fee Paid 



SUBTOTAL (3) 



($)_ 



SUBMITTED B> 


f 


Compiete (if applicable) 


Typed or 
Printed Name 


V. Gerald Grafe 


Reg. Number 


42,599 


Signature 




Date 




Deposit Account 
User ID 





Burden Hour Statement: This form is estimated to take 0.2 hours to complete Time will vary depending upon the needs of the individual case. Any comments on the amount of 
time you are required to complete this form should be sent to the Chief Information Officer. Patent and Trademark Office, Washington, DC 20231 . DO NOT SEND FEES OR 
COMPLETED FORMS TO THIS ADDRESS. SEND TO: Assistant Commissioner for Patents, Washington, DC 20231 . 



SD6337/S92307 
October 24, 2000 



FRACTIONAL SCREEN VIDEO 
ENHANCEMENT APPARATUS 

INVENTORS: 



Barry L. Spletzer 
9504 Arvilla, N.E. 
Albuquerque, New Mexico 87111 



George 8. Davidson 
1517 La Tuna, S.E. 
Albuquerque, New Mexico 87123 



Daniel J. Zimmerer 
42 Campo Rd. 
Tijeras, New Mexico 87059 



Lisa C. Marron 
405 Mankin St., N.E. 
Albuquerque, New Mexico 87123-1124 



Express Mail No.: EL027381 824US 



SD6337/S92307 

Spletzer 



Fractional Screen Video Enhancement Apparatus 

This invention was made \N\th Government support under Contract DE-AC04- 
94AL85000 awarded by the U. S. Department of Energy. The Government has certain 
rights in the invention. 

Background of the Invention 

This invention relates to the field of displays, specifically displays using multiple projectors to 
provide enhanced resolution. 

Conventional displays can be characterized by their resolution. Generally, higher resolution 
means higher display system cost. In many display applications, however, higher resolution is 
needed for only portions of the overall image. For example, in display of a sporting event, only a 
limited region (e.g., the pitcher and batter in a baseball game) benefits from high resolution at any 
given time. As another example. In a display of a computer image, only the portion currently being 
inspected needs to be displayed in high resolution. In both examples, low resolution display of the 
surrounding context can aid understanding of the portion displayed at high resolution. Also, the 
portion that benefits from high resolution display can change with time (the batter hits the ball and 
begins running the bases in the baseball game example). A conventional display with sufficient 
resolution for the high resolution portion can be more expensive; a conventional display at lower 
cost can be incapable of the high resolution needed for some portions of the image. 

Accordingly, there is a need for a display method and apparatus that can display portions of an 
image at a higher resolution and the rest at lower resolution, without incurring the cost associated 
with displaying the entire image at the higher resolution. 
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Description of the Figures 

The accompanying drawings, whicln are incorporated into and form part of tlie specification, 
illustrate embodiments of the invention and, together with the description, serve to explain the 
principles of the invention. 
5 Figure 1 is a schematic representation of a video display system according to the present 
invention. 

Figure 2 is a schematic representation of a video display system according to the present 
invention. 

Figure 3 is an illustration of geometrical relationships relevant to the present invention. 
^0 Figure 4 is a schematic representation of image projection associated with the present invention. 

^3 Detailed Description of the Invention 

H The present invention provides a method and apparatus for displaying two portions of an 

a image at two resolutions. For example, the invention can display an entire image at a first 
resolution, and a subset of the image at a second, higher resolution. Two inexpensive, low 

r|5 resolution displays can be used to produce a large image with high resolution only where needed. 
Figure 1 is a schematic representation of a video display system 10 according to the present 

'"^ invention. Video display system 10 comprises first 1 1 and second 12 video sources, arranged 
relative to a display medium 14. First video source 1 1 accepts input corresponding to a first 
portion 16 of an image to be displayed. An image transformer 1 3 accepts input corresponding to a 
20 second portion 1 7 of the image to be displayed and information defining the configurations of the 
first 1 1 and second 12 video sources and the display medium 14. Image transformer 13 
generates a signal input to second video source 12 compensated for the configurations such that 
the resulting display of the second portion on the display medium is aligned with the display of the 
first portion on the display medium. 
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First AND Second Portions of the Image 

The first and second portions of the image can be any portions of the image, including the 
entire image. For example, the first portion can be an entire image to be displayed, and the 
second portion can be a subset of the entire image that requires display in higher resolution. The 
5 first portion can be, for example, a display of a camera image while the second portion is a subset 
of the camera image deserving specific attention (e.g., where motion occurs, or where detailed 
features are present). The first and second portions can change with time. In the entire 
image/subset example, the entire image can change with time, as with a panning camera or an 
animated computer display. The subset can also change with time, both in content (as the image 

4-0 itself changes) and in relation to the entire image (as the subset deserving attention moves 
relative to the image). Motion of the second portion relative to the entire image can follow, as 

"^i examples, a viewer's eye movements, motion of a pointing device, and motion of an onscreen 

H object (such as a ball in a sporting event). 

= Display Medium 

rt5 The display medium can be of any form compatible with the video sources. As examples, rear 
j; projection displays, screens, and suitable wall surfaces can each serve as a display medium. 

First Video Source 

Those skilled in the art know a variety of suitable video sources suitable for use in various 
settings. Any combination of display technology that would be capable of displaying 2 different 

20 resolutions on the same screen may be used. Examples of suitable image display sources include 
rear projection systems, cathode ray tube (CRT), LCD, Digital Light Projectors (DLP), and plasma 
displays. The first video source can be operated in a conventional method to display a larger, 
lower resolution image. The mounting and alignment of the two video sources depends on the 
type of display sources, optical properties (such as focal length and depth of focus), image 

25 steering apparatus, and the geometry of the system. 
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In the example implementation described below, the two projectors were rigidly mounted to a 
base to reduce the degrees of freedom of the system. The main projector was set approximately 
perpendicular to the display screen to limit distortion of the main image. However, this is not 
necessary. The transform (TJ compensates for linear distortion in overall transform. If the 
5 projection surface (screen) is planar, then homogenous transforms can be used. Otherwise, more 
complex transforms can be required. 

The second projector was mounted so that the focal range would coincide with the display 
screen after reflection from the mirror, and such that the optical axis of the lens intersects the 
mirror coordinate system at the origin. The angles between the projectors are arbitrary, and were 
10 chosen to restrict the overall size of the apparatus. 

m Second Video Source 

1^ The second video source can be any of a variety of video sources as discussed above. The 
second video source can be mounted so that its display corresponds to only a subset of the 
region covered by the display of the first video source. If the second portion of the image is mobile 

tJS with respect to the first portion, then the second video source must be able to adjust the 
placement of its display relative to the display of the first video source. 

D A conventional video source can be combined with an image steerer to provide the second 
video source. The image steerer directs the output of the conventional video source to the 
appropriate portion of the display medium. As examples, an image steerer can comprise a 

20 moveable mirror, a controllable optical transmitter, or a micro-electromechanical device. A 
conventional video source can also be mounted mobile with respect to the display medium, 
allowing the output to be directed to the appropriate portion of the display medium by movement 
of the video source. 

Image Transformer 

25 The second portion of the image travels a different path to the display medium than does the 
first portion. Accordingly, the display of the second portion can be adjusted to achieve proper 
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registration between tine two portions. In the discussion below, it is assumed that the second 
portion is directed toward the display medium by a gimbal-mounted mirror, controlled by computer 
commands to stepper motors, as shown in Figure 2. Those skilled in the art will appreciate similar 
image transformations suitable for other mechanisms directing the second portion to the display 
medium. 

The output of the second video source can undergo significant distortions depending on the 
location on the display medium to which it is projected. These distortions can include skew, 
rotation, and anisotropic stretching. The second portion can be transformed before reaching the 
second video source so that, after all the distortion, the display on the display medium is properly 
registered. The transformation can be complex, but, since ray tracing from the object plane of the 
second video source to the display medium involves classical projection, rotation, and reflection, 
the transformation can be expressed as a single homogeneous transform. Development of such 
transforms is discussed below. 

Basic homogeneous transforms 

Homogeneous transforms can provide a powerful mathematical tool for converting between 
coordinate systems and determining the projection of images. A homogeneous transform 
comprises a 4x4 matrix that operates on a four element homogeneous coordinate vector. The first 
three elements of the homogeneous coordinate vector are the usual x, y, and z components used 
to describe a three-dimensional point. The fourth component (d) allows projection information to 
be carried along with the three-dimensional point. For a point In actual space this fourth 
component is always equal to 1. The usefulness of this fourth component and its physical 
meaning will become more clear later. 

Homogeneous transforms are true matrices. Accordingly, they are amenable to various matrix 
operations; for example, several simple transforms can be multiplied together to produce a more 
complex transform. A homogeneous transform can be understood by examining some of the 
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basic components that can contribute to the transform. The first of these is translation of the 
coordinate system origin. An origin translation transform has the form shown in Equation 1. 

Equation 1 
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The transform of Equation 1 converts a general point in the s coordinate system to the b 
coordinate system. Examining the operation shown in Equation 1, the general point in the vector t 
is simply added to the point expressed in the s system. This constitutes a translation transform 
where the vector t is defined as the vector from the origin of the ib system to the s system. Rather 
than writing on all the elements explicitly, this translational homogeneous transform can be 
defined as in Equation 2. 

fl 0 0/1 

Equation 2 
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In addition to translation, rotation between two coordinate systems can be defined by the 
^tt) homogeneous transform of Equation 3, where C is a point expressed in the c coordinate system 
y and B is the identical point expressed In the b coordinate system. 

Equation 3 
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All of the elements of this rotation vector are made up of dot products between the unit vectors 
of the b and c coordinate systems. The notation used here for this rotational homogeneous 
transform is shown In Equation 4a. 

Equation 4a 
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The rotational transform and origin translation transform can be combined to perform a 



complete coordinate system transform as in Equation 4b. 



C(b,c) = 0{b,c)Rib,c) 



Equation 4b 



In addition to rotation and translation scaling may be accomplished by a homogeneous 
transform. Simple isotropic scaling is done by multiplying each of the three components by a 
5 fixed scale factor f. The scaling transform is defined as in Equation 5. 



0 0 0 1 

□ The next basic transform is a projection transform which uses the bottom row of the 

tn homogeneous transform. Before presenting the form of this transform, some explanation of the 

Ul fourth element (of) of a homogeneous coordinate can be useful. 

f 5 As mentioned before, a point in space can be represented by the usual xyz coordinates and a 

jb d value of 1 . The definition of homogeneous coordinates states that any four element coordinate 
can be converted into a point in space by dividing all four elements by the value of of. By defining 
this as a valid operation on a homogeneous coordinate the ratios between the first three elements 

^::f are invariant. This means that a homogeneous coordinate describes a line passing through the 
origin from which the point is measured and through the actual point in space. This property 

15 makes homogeneous coordinates very useful for calculating projections. If the origin of the 

coordinate system is moved to the observation point or perspective point then the homogeneous 
coordinate of a point defines a line that passes through the point in space and through the 
observation point. By definition, this line is the projection of the actual point through the 
observation point. 

20 Using the projection transform, the homogeneous representation with respect to the origin can 
be produced for any point. By adjusting the fourth coordinate, any point along the line passing 
through the original point and the origin can be represented. Specifically, consider the value of the 



S(/)- 



/ 0 0 0 
0/00 
0 0/0 



Equation 5 
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fourth coordinate required to represent the point where the line intersects a plane. The plane is 
completely defined by the vector N that connects the origin to the image plane and is normal to 
the image plane. Figure 3 illustrates the geometry of this situation. The original point C is 
measured with respect to the origin placed at the observation point. The vector Cp is the 
projection of the point onto the plane. 



Similar triangles lead to Equation 6. 

C-N 



C 



N C„ 



The vector C can be scaled to represent Cp as in Equation 7. 



Equation 6 



C N N-N 

c -c— -c — — = c 

P C C-N C'N 



N 



Equation 7 



V N-N^ 



By the definition of homogeneous coordinates, the fourth coordinate (of) can be divided into the 
homogeneous representation to create a point in space. Therefore, to convert the point C to the 
point Cp, only the value of d need be modified. The value of cf required to perform the conversion 
is given by Equation 8. 

C-N 

d^TT-rz^C'M Equations 



N-N 

The vector M can be completely determined from the projection plane definition N as in 
Equation 9. 



M 



N 
N-N 



Equation 9 
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The bottom row of the homogeneous transformation can be used to produce this value. The 
first three components of the bottom row operate on the vector to produce a dot product. The 
fourth component must be 0 so that the proper value of of is obtained. This results in a transform 
of Equation 10, which is the complete projection transform. 

Equation 10 



P(n) = 
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The final basic transform is the zoom transform, which consists of the identity matrix with a 
zoom value in the lower right element. The zoom value is a multiplier of the input d value of the 
homogeneous coordinate. Since the d value can be divided into the homogeneous coordinate to 
produce a point in space the zoom factor is the isotropic scaling of the distance from the origin to 
the input point. After the transform, the distance to the point is 1/z times the distance to the 



original point. The zoom transform as defined as in Equation 11. 

1 0 0 0' 



Z{z) = 



0 10 0 
0 0 10 
0 0 0 z 



Equation 11 



One more transform is needed to completely analyze the system. This is the transform 
resulting from reflection in a mirror. The nature this transform will be determined here based on a 
vector analysis of reflection. The position of the mirror can be completely described by the normal 
vector (n) to the mirror. In general this normal vector will be known and the resulting transform is 
desired. Unlike previous transforms which expressed a given point in a second coordinate 
system, the mirror transform takes a point in the original coordinate system and expresses 
location of the reflected image of that point in the original system (virtual point) 

The development of a transform to handle mirror reflection requires the determination of the 
relationship between the incident coordinate system and the reflected coordinate system. To 
simplify this determination, place the origin of both coordinate systems coincident on the mirror. 
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The z-axis (k) of the incident system points towards the mirror. The required rotation to produce 
the reflected system must be determined. The notation for the reflected system will be to add a 
prime {) to the notation of the incident system. The law of reflection gives Equation 12. 

k'^k-(2k-n)n = k- In^n Equation 1 2 

The equation of the mirror plane is given in Equation 13, where A is any vector lying in the 
5 mirror plane. 

A.?5 = 0 Equation 13 

To determine the direction of the reflected x-axis, consider the projection of a ray parallel to the 
z-axis but displaced one unit in the x direction. The intersection of this ray and of the z-axis with 
the mirror creates two points, which define the projection of the x-axis onto the mirror. The vector 
yi representing this projection can be determined by noting the equation of the incoming ray, given 
ylD in Equation 14. 

ffj X = 1, y = Q Equation 14 

u3 Solving for the z coordinate of the intersection of the ray with the mirror plane yields 
Equation 15. 

[i+zk)'n^O Equation 15 



The projection of the x-axis onto the mirror (/J has the above x, y, and z components as in 
Equation 16. 



1^ = 1- — k Equation 16 



15 This vector, when viewed from the /c'direction, constitutes the x-axis. The vector can be 
projected into the plane perpendicular to the /c'direction by subtracting the component of the 
vector parallel to k'. This results in the expression for /In Equation 17. 

^' = L-[L-k'}k' Equation 17 

Equation 17 shows /'as a unit vector. This is true because an object is not foreshortened by 
reflection So that the magnitude of the reflected vector is the same as the incoming vector as long 
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as the view direction relative to the vector remains the same. Combining this expression with the 
above relationships gives Equation 18. 



m 



n. 



■k' 



k' 



Equation 18 



= i -—k-[i •k')k' + 



n ^ ^ 



Evaluating some useful subexpressions gives Equation 19. 

k - {ik • njn] - -(ik • nj{i - nj 



i 'k' -i ' 
^k' 



Equation 19 



k -{2k ^n\n\ = l-2[k 'fi 



Combining yields Equation 20. 

i'=i--^k + 2[k-n]{i •n)k' + -^\-2[k-n) 



k' 



Equation 20 



i' =i- ^2(k ■ njn + 2{k ■ nU ■n-[k-n)^\k' 



i'=i-'^2(k-n]n + l(k-n] i-n-ik-n]'^ 

n, ^ ' ^ \ ^ ' n 



k — \2k -nm 



^ ^ ft 

i' - —2nn + 2n 



- 5 Since all the indicated dot products with the mirror normal have one of the original unit vectors 
O as a component, they can be replaced by the components of the mirror normal unit vector as in 
^ Equations 21 and 22. 

k-[2n^)n\ Equation 21 

i' —i - 2n^n 

f ^j-2n^n Equation 22 

From this, the homogeneous transformation matrix of a mirror can be developed. This matrix 
consists solely of rotation with the origin of both systems located coincident on the mirror surface 
10 as shown in Equation 23. 

Equation 23 



M(n) = 



l-2n, -2n^ny -2n^n^ 0 
-2n^ny l-2nl -2«^n^ 0 
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The notation and description of eacli of the required contributing transforms is summarized in 
Table 1 . 

Table 1 



Transform 


Symbol 


Description 


Origin translation 


OM 


Translates coordinates from origin of c system to origin of b 
system 


Coordinate rotation 




Converts coordinates from c system to b system provided 
that the system origins are coincident 


Coordinate conversion 


C(6,c) 


Convert coordinates from c system to b system accounting 
for both rotation and translation 


Isotropic scaling 


S(/) 


Isotropically scales coordinate values by factor f 


Projection 


P(N) 


Projects points from three-dimensional space onto an image 
plane where N is the vector from the coordinate system origin 
to the image plane and normal to the image line 


Zoom 


Z(z) 


Scales three-dimensional points so the resulting point lies 
along the original line connecting the origin with the point but 
at a fraction 1/z from the origin 


Mirror reflection 


M(n) 


Determines the image of the original point has seen in a 
mirror based on the unit normal n to the mirror 



Image projection using homogeneous transforms 

Before tackling the more difficult problem of two projectors and a mirror, consider the problem 
of producing a projection of an object onto an image plane. This requires that all rays from the 
object to the image plane pass through the pinhole or observation point. The intersection of such 
a ray with the image plane is the projected image of the original point. Figure 4 shows this 
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schematically. A single honnogeneous transform can be constructed to perform this operation by 
combining several of the previously discussed basic transforms. 

In the figure, the points in the object are originally expressed in the S coordinate system. The 
homogeneous coordinate representation of the general vectors is given in Equation 24. 

Equation 24 



1 

5 The 1 for the fourth element means that the actual values of the vector S are used here. First, 
construct a simple rotation and translation transform to convert from the S coordinate system to 
the C coordinate system with the origin at the pinhole. This transform is in Equation 25. 
yi C-C(c,^)S Equation 25 

"^3 in this transform, c and s refer to the appropriate unit vectors of the two coordinate systems 
M and So is the vector from the origin of c to the origin of s measured in the c coordinate system. 
5l) Using this transform, the homogeneous representation with respect to the pinhole can be 

produced for any point in the original object. By adding a projection transform the coordinates of 
ry any point in the image plane can be determined, as in Equation 26. 

1 Cp=P(N)C(c,^)S Equation 26 

This gives the coordinates of all points projected onto the image plane in the pinhole 

coordinate system. One final conversion is needed and that is to convert the coordinate system 
15 reference to the image plane {b). This transformation has the same form as the transformation 

between the s and c systems earlier. Adding the appropriate rotation and translation transforms 

results in Equation 27. 

B = C(&,c)P(N)C(c,5)S Equation 27 

This result provides the homogeneous coordinates in the image plane coordinate system. In 
general the value of the fourth element (cf) will not be 1 .0 and all elements in the resulting vector 
20 must be divided by d to yield the actual point in space. 
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Experimental determination of homogeneous transforms 

The previous derivation shows that a single homogeneous transform does exist to perform the 
projection. However, since it is constructed from many basic transforms multiplied together, the 
relatively simple components that make up each transform and the usual association of areas of 
5 the transform with rotation, translation, perspective, and zoom will in general be lost. The overall 
transform can be determined experimentally without any advance knowledge of the contributing 
parameters. Equation 28 gives the completely general form of a homogeneous transform. 
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Equation 28 
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15 



Point-by-point measurennents in the s and b coordinate systems can be made to 
experimentally determine the transform for a given system. Each point measurement will yield 
three new scalar equations but introduce one additional unknown, that being cf which is unique for 
each point. Because of this, each measured point will provide information to solve for two 
additional unknowns. The set of equations resulting from these measurements is homogeneous. 
However, each element in the transformation matrix can be divided by a constant without altering 
the result. This is because of the scaling factor cf. Dividing each element by a constant divides the 
resulting value of d and the values of b^d, byd, and b^d by the same constant so that the 
determination of the final coordinate is unaffected. Doing this results in the set of linear equations 



,10 for each measured three-dimensional point shown in Equation 29. 
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Equation 29 



The 16th value of the transformation matrix, is set to 1 .0. A set of these three independent 
equations can be constructed from any measured point where a coordinate in the S system is 
measured and corresponds to a known point in the B system. This way, in the general case, at 
least five points can be measured to completely determine the transformation. Since all the 
equations are linear with respect to the unknown coefficients, linear regression 
techniques (discussed later) can be used to reduce the error in the resulting transform by 
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measuring additional points. In tlie specific case of interest, points in both the S and the B 
coordinate systems are constrained to a plane so that and s.are always zero. This reduces the 
equations to be solved to Equation 30. 



1 0 0 0 -Z^A -K^ 



0 0 0 5, 
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Equation 30 



This set of equations provides solutions for only eight of the required 15 components. The 
seven undetermined components are used only as multipliers of the z components or to 
determine the z value of component. This set of equations shows that a minimum of four points 
can be required to completely determine the transform between the two planar coordinate 
systems and, as before, additional points can be used to provide a best fit transform. 

The use of only a minimum number of points can allow small errors in measurement to 
introduce very large errors in the resulting transform. Measuring many extra points and 
performing linear regression on the data can reduce the errors. The determination of the 
homogeneous transform via linear regression can proceed as follows. For each point measured 
on the system the left and right-most matrices can be augmented by adding to new rows. For 
example, if only three points are measured than the resulting equations set is as in Equation 31 . 

Equation 31 
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Equation 32 represents this symbolically. 

s t= b Equation 32 

The solution for the transform elements t using linear regression is as in Equation 33. 

t- (s^s)''(s'^b) Equation 33 

This solution for the transform works equally well for the two-dimensional case where the eight 
elements of t are solved or for the three-dimensional case that solves for all 15 elements. In either 
5 case, U4 can be set equal to 1 without any loss of generality. 

Construction of the complete transform 

Combining the ability to experimentally determine the homogeneous transform for overall 
f i system with the property of combining basic transforms to produce a complex transform allows 
fi the needed transforms to be constructed. The method can be complex because the movement of 
fp the mirror alters the transform in a complex way. 

The determination of the complete transform by measuring individual points can be done for a 
W transform between the two projectors just as well as from one projector to the screen. Several 
intermediate coordinate systems can be convenient for this analysis; these are shown in Table 2. 

■ : Table 2 



Symbol 


System description 


Alignment 


b 


small image projector object plane 


z normal to object plarie 


L 


large image projector object plane 


z normal to object plane 


s 


Projection screen 


z normal to screen, origin along the mirror z 
axis, parallel to and parallel to 


P, 


small projector pinhole 


Unimportant 


Pl 


large projector pinhole 


Unimportant 


m 


mirror system 


z parallel to screen normal, origin at mirror axis 


Pv 


small projector virtual pinhole 


Unimportant 
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The reasons for the various coordinate system alignnnent will be explained in this section. The 
techniques for achieving this alignment will be explained later. 

For the final system a number of calibration points can be measured to determine a specific 
transform. This is not sufficient since the transform is a function of the mirror position. Instead, the 
transform resulting from the calibration points must be decomposed to allow subsequent 
transforms to be constructed as a function of mirror position without additional measurements 
being required. The complete transform at any position can be expressed as a product of three 
transforms. The first of these three is the right transform (Tr), in Equation 34, which transforms 
from the small projector object plane to the mirror coordinate system. 

= C(m, P^)C{P„,b) Equation 34 

This is a direct coordinate transformation where all coordinate systems are fixed relative to 
each other so the transform is constant. The second is the center transform (Tc), in Equation 35, 
which converts from the mirror coordinate system to the screen. 

Tc = C{s,m)0{m, i'JP(N)0(P„m)M(n) Equation 35 

This transform requires some explanation. Going through the steps in sequence from right to 
left, the transform begins with a point on the small projector object plane expressed in the mirror 
coordinate system. Applying the M{n) transform gives the coordinates of the image of the point as 
seen in the mirror. The next transform, 0(P„m) translates the coordinate system back to the 
virtual pinhole. From here, a projection transform P(N) converts the original point to the 
coordinates of the point projected onto the screen. The vector N is defined as the normal vector to 
the screen from the origin of the virtual pinhole system which is the virtual pinhole itself. Part of 
the difficulty in determining this center transform is the fact that the position of the virtual pinhole 
changes with changing mirror position. After the points are projected, the origin translation 
transform 0(m,PJ converts back to the mirror coordinate system. The final basic transform 
C{s,m) converts the projected points into screen coordinates. 
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The final transform is tine left transform(TL), in Equation 36, winich converts from the screen to 

the large projector object plane. 

\ - C(L,P^)C{P^,s) Equation 36 

This is similar in form to the right transform and is a coordinate transformation between two 

fixed systems. Here however the constant value of the transform is dependent on the screen 

5 position relative to the projector box. Unlike the right transform which need only be measured 

once for a projector configuration, this transform can need to be determined for each screen 

setup. 

This division into three transforms is chosen because the center transform contains all the 
variables resulting from the repositioning of the mirror and the remaining transforms are constant 
JlSO for a given screen and projector position. The total transform can be measured for any mirror 

position, as represented by Equation 37a. 
y T^easur^d = T^T^ (ii)Ti, Equatlon 37a 

^2 For any given value of the mirror normal (n) the measured transform can be determined. It is 

- desired to determine the constant value of both the left on right transforms and to discover a way 
Q to construct the center transform from the value of the mirror normal vector. Rearranging 
J35 Equation 37a yields Equation 37b. 

□ ^easured^"' = \^c(^) Equatlon 37b 

For a given screen setup the left transform can be determined empirically as described 
previously. Also, for a given known mirror position, the measured transform can be determined. 
Once this is done, the above matrix equation becomes 16 equations with 32 unknowns since 
each element on the left side of the equation is a linear function of the unknown elements in the 
20 inverse of the right transform and each element on the right side is a linear function of the 
unknown elements of the center transform. 

By moving the mirror and re-determining the transform no additional unknowns are introduced 
on the left side of the equation but a completely unknown center transform results. Therefore, 
each additional mirror position measurement adds 16 equations and 16 unknowns. Because of 
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this the center transform must be examined more carefully to reduce the total number of 
unknowns. The right-most contributing transform to the center transform of Equation 35 is a mirror 
transform, which Is known by the position of the mirror measured in the mirror coordinate system. 
The next transform is an origin translation which consists generally of three unknowns. However, 
the magnitude of the translation is the distance between the mirror and virtual pinhole coordinate 
systems which is equal to the distance between the mirror and the actual pinhole. Therefore, the 
magnitude of the translation is constant for all mirror and screen positions. Calling the pinhole to 



mirror origin vector p the transform becomes as in Equation 38. 

l 0 0 



0 I 0 
0 0 I p, 
0 0 0 1 



Equation 38 



This means that this contributing transform only has two additional unknowns for each mirror 
position. The third transform is the projection. By the alignment of the mirror coordinate system, 
the screen normal vector N is parallel to the mirror coordinate z-axis so the transform can be 



represented by a single unknown, as in Equation 39. 

10 0 0 



P(N) 



0 10 0 
0 0 10 

0 0^0 



Equation 39 



The next transform moves the origin back to the mirror coordinate system. Since the 
operations following the previous translation do not affect the orientation or scaling of the 
coordinate system, this translation is the negative of the previous one, and is shown in 
Equation 40. 

fi n n _ n 1 

Equation 40 



1 0 0 

0 10-;^, 

0 0 1-/?, 

0 0 0 1 
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The final transform converts between the mirror and the screen coordinate system. The 
stipulation that the mirror and screen coordinate systems are aligned considerably simplifies this 



transformation. The result is Equation 41. 

-10 0 0 



C(s,m) = 



0 10 0 
0 0 -1 
0 0 0 1 



Equation 41 



In Equation 41 , it is the distance between the two coordinate systems. Multiplied together 



5 these contributing transforms result in Equation 42. 
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0 1 
0 0 
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Equation 42 



M(ii) 



Noticing that N = p^+z^ yields Equation 43. 

p. P.^0 



Tc(n) = 



-1 0 



0 1 - 

0 0 
0 0 



Py 



P.+^o 
Py^o 



P.+^o P.+^o 

0 0 

1 p. 



Equation 43 



M(ii) 



The values of p^, Py, and change with mirror position. However, the values can be 
transformed from the virtual pinhole to the real pinhole by Equation 44. 

P = M(ii)p Equation 44 
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In Equation 44, P is the fixed vector from the origin of the mirror system to the actual pinhole 
and p Is the vector to the virtual pinhole. Performing the transform and explicitly substituting the 



values into the center transform yields Equation 45. 



Tc(n) = 



-1 0 



0 1 - 

0 0 
0 0 



Py - 2«,P • n 



0 
1 



P,-2n,P-n + z„ 
0 

F,-2«,P-n + z, ^ 



Equation 45 



M(ii) 



At first glance this seems to have complicated the transform. This is not the case. The previous 
transform had three unknowns which vary with mirror position. The unknowns in this transform 
are all invariant with mirror position. This means that the determination of the total transform at 
multiple mirror positions introduces no additional unknowns and therefore the unknown fixed 
values comprising the three contributing transforms can be determined with sufficient 
measurement. Because the relationships between the unknowns do not result in linear equations, 
the determination of P and Zo is still not a trivial task. 

The values for P and Zq can be determined by using a two-step linear least squares approach. 



First define the unknown matrix above as in Equation 46. 
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Equation 46 



= A 



Now the transformation takes on the form of Equation 47. 

-10 -A^o 



rp— Inn rp — 1 
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0 1 -X 
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Equation 47 



M(ii) 
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As discussed before Tjt can be measured using the large format projector and any suitable 
screen. For any given mirror position (n) Treasured is determined by correlating positions on the two 
projector images as measured at the screen. This means that the remaining unknowns are the 
elements of Tr-\ A,, A^, A3, A„ and Zofor a total of 21 unknowns. The transform Tr^ is invariant ' 
with mirror position but A is depending on the mirror position. This means that a sufficient number 
of equations can be generated by determining Treasured at a suitable number of mirror positions. 
Each mirror position produces 16 equations. However, since each position only adds four 
unknowns {A^, A^, A3, A^^ to the system, measurement of as little as two mirror positions can be 
sufficient to perform a least squares analysis and determine all the needed values. 

The use of multiple mirror positions gives multiple values for A, but the linear regression will 
determine the A matrix for each mirror position. Once the family of A matrices are determined, a 
second linear least squares can be used to determine the values of P and Zq. This can be done 
because the determined elements of A can be combined with the algebraic definition of the matrix 



to construct a series of linear equations. These linear equations are in Equation 48 

\-2nl -2k,«„ -2n/i^ 0 



X X y 
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-2n^n, -2n,n^ \-2nl 1 



P. 
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4 
1 



Equation 48 



These equations can be used similarly to the technique described in empirical determination of 
a homogeneous transform to construct a linear least squares fit to the unknown values. 

Standard data analysis techniques may be used to calculate the regressions and transforms. 
Computer programming languages (C, basic, etc) or engineering calculation software (Mathcad, 
Matlab, etc) can be useful tools in this calculation. The display hardware determines the 
parameters for possible graphics software packages used in a particular system. For our 
example implementation LCD projectors were used, so any input compatible with the projector 
was sufficient. If the 2"" projector were mounted to a pan and tilt platform, similar transfomations 
could be used. The transforms could be derived using a platform coordinate system instead of 
the mirror coordinate system. A similar system of matrices would result, with the absense of the 
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mirror reflection transformation and the inclusion of tlie motion platform coordinate conversion 
transformation. 

Other configurations could be similarly derived. Consideration should be given to choice of 
coordinate systems, alignment of coordinate systems, degrees of freedom of the system, choice 
5 of steering system, and transformations of the optical paths of the images. If these system 
parameters for a particular setup are chosen carefully, the homogenous transforms can remain 
relatively straightforward. Additionally, the choice of lenses used with projection systems can be 
important. It is desirable to minimize radial or other non-linear distortion in both images. 
Nonlinear transformation schemes exist to compensate for some these distortions, but they can 
1 0 be difficult to implement. 

S Example Implementation 

[S A system according to the present invention is depicted in Figure 2. The system used two 
f5 Proxima (trademark of Proxima Corporation) 9200 projectors, each having a maximum image 
resolution of 1024x728 picture elements. See "Proxima Desktop Projector 9200 User's Guide", 
3^ Proxima Corporation. One projector was fitted with a short focal length lens; the other with a 
-^^ variable focal length zoom lens. The longer zoom lens was supported to prevent movement of its 
y optical axis. 

A Model PTU-46-17.5 pan and tilt unit from Directed Perception, Inc., modified to provide 
access to the line where the pan and tilt axes intersect, was used to direct the high output of the 
20 second projector. See "Pan-tilt Unit User's Manual", Directed Perception, Inc. That model has a 
resolution of 0.0514° and can achieve speeds of 300"" per second. The pan-tilt unit was controlled 
by a computer through a standard serial port. 

The second projector's optical axis was directed towards the center of rotation of the mirror. An 
aperture was used to block some of the high intensity light and provide a longer depth of focus for 
25 the long focal length lens. A 1^" diameter aperture was placed between the 1®* and 2""^ lenses 
inside the zoom lens assembly. This provided an adequate depth of focus. However, minor 
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interference patterns and color distortion can be present in this setup. Neutral density filters can 
be used to further reduce the intensity of the high resolution image. Two video inputs are 
necessary for the system, one for the main projector and one for the enhanced image. For an 
initial prototype and testing, two VGA laptop computers were used to produce the images and 
5 control the mirror assembly. The tested resolution was set at 640x480 pixels (due to the laptop 
capabilities) on each computer and projector. At this resolution, the projectors could be set to the 
expanded display setting. 

Calibration 

Those skilled in the art will appreciate various methods for determining the values suitable for 
Bp the transform. One example, used with the example implementation above, is described below. 
S Drafting vellum with 0.1 inch grid can be applied to a hard surface screen as a background for 
m the images and a means of recording data. The screen can be positioned so that it is 
O approximately parallel with the front edge of the system, and then aligned parallel to the mirror 

surface. During the calibration of the system, the display screen can be aligned with the front 
Sfc surface mirror using a visible laser calibration tool. A laser can be mounted in a block such that 

the light emitted is perpendicular to the bottom surface of the block. The system can then be 
O adjusted by placing the bottom of the block on the flat screen, and adjusting the screen (or 

projector apparatus or mirror) so that the display screen is parallel to a mirror position (when the 

laser reflects back upon itself). The position of the mirror at this alignment can be recorded, and 
20 then used to create the fixed mirror coordinate system. The pan-tilt can then be rotated 90° about 

the pan axis, and the point where the tilt axis intersects the screen located. This defines the 0,0 

screen coordinate for measurement of the data. 

A grid of data points (spaced at 40 by 40 pixels over the entire image) can be projected to the 

screen through the mirror, and to the screen from the main projector. Each grid point can be 
25 recorded on the vellum, and the mirror position noted. Once a few mirror position grids and the 

main grid are recorded, the physical x,y coordinates of each grid point can be written into a data 
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file along with the pixel location of each point. The amount of grid points is much greater than 
actually needed, but provides an over determined system for the linear regression technique for 
finding the measured transform from the measured data. The transform can then be found using 
the method described in the above. 

If P and Zo can be measured or estimated, then the transform can be determined without a 
double regression. The long focal length of a zoom lens on the second display source allows for a 
"pinhole" projection model to be used. An approximation for the focal distance of the lens can be 
calculated using Equation 49. 

— = — ; /i + A = ^0 + Equation 49 

^1 fi 

In Equation 49, w^ is the width of the LCD panel inside the projector, W2 is the width of the 
image on the screen, f^ is the distance from the pinhole to the LCD, fs is the distance from the 
pinhole to the screen, Zq is the distance from the mirror center of rotation to the screen and z^b "s 
the distance from the mirror center of rotation to the LCD panel (the magnitude of the constant P 
vector). Solving this system for f^ can provide a reasonable measure of effective focal length of 
the lens. 

Next, the "pinhole" vector P and the orientation of the enhanced image LCD can be found 
utilizing one position of the mirror and ray tracing. If it is assumed mathematically that the z axis 
of the screen and the LCD intersect at the mirror center of rotation, the screen coordinate of the 
x,y pixel located on the optical axis can be found in mirror coordinates. Reflecting this normalized 
vector from the mirror provides the unit direction of the optical axis. Since the magnitude is 
known, the P vector is now known. The screen coordinate of the pixel representing the maximum 
X value on the LCD can be used to determine the orientation of the x axis in mirror coordinates. 
Thus, the orientation of the LCD can be determined. 

From this, the T^ matrix can be created using the equation Tr=ABC where C consists of a 
scaling and translation matrix to transform computer coordinates into inches, B is the rotation 
from the LCD system to the mirror coordinate system and A is translation from the LCD origin to 
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the mirror coordinate system origin. Tliis is the complete transform from the LCD to mirror 
coordinates, and is dependent only on the pinhole. This transform remains constant for a given 
screen setup. 

The next part of the transform is created using Equation 50. 

=C(s,m)0(m,Pv)P(N)0(Pv,M)M(n) 

Equation 50 

Components of Equation 50 are given by Equation 51, where p^ is the location of the virtual 
pinhole in mirror coordinates, Zo is the distance to the screen, and n is the mirror normal vector in 
mirror coordinates. A new transform can be created for each mirror position,. 
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Multiplication of this transform by the Tl matrix (measured transform from screen coordinates 
to main screen computer coordinates) completes the transformation from enhanced image 
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computer coordinates to main screen computer coordinates. Inverting this completed transform 
provides the transform from main screen computer coordinates to enhanced image coordinates. 

In order to calculate where the enhanced image should be drawn, the mirror normal can be 
found based on the screen coordinate that the axis pixel should be located. Knowing this point 
and the location of the pinhole in mirror coordinates, the mirror normal can be found by 
subtracting the normalized vector to the pinhole from the negative of the unit vector to the screen 
position (all in mirror coordinates) and then normalizing the result. The pan and tilt steps to 
produce this normal can be found by solving the following system of equations. 

The relative angles to the mirror coordinate system can be calculated using Equation 52. 

-n^=sin(0p)xos(et) Equation 52 

-n =sin(0t) 

y 

-B^=(cos(0p)*cos(et)) 

In Equation 52, Gp = relative pan angle in mirror coordinates and Gt = relative tilt angle in 
degrees in mirror coordinates. The pan and tilt encoder steps can be found by Equation 53, 

where pano and tiltp are the encoder steps used to define the mirror coordinate system.. 

pan=-(0p.lll4.O849-panJ Equation 53 

tilt=0t- 11 14.0849- tilt^ 

The particular sizes and equipment discussed above are cited merely to illustrate particular 
embodiments of the invention. It is contemplated that the use of the invention may involve 
components having different sizes and characteristics. It is intended that the scope of the 
invention be defined by the claims appended hereto. 
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Claims 

We claim: 

1 . A video display system for displaying on a display medium an image, wherein the image 
comprises a first portion to be displayed at a first resolution and a second portion, mobile 
with respect to the first portion, to be displayed at a second resolution, comprising: 

a) A first video source mounted relative to the display medium; 
5 b) A second video source mounted relative to the first video source and the display 

medium; 

c) An image transformer generating an input to the second video source such that the 
second video source displays the second portion aligned with the first portion 
displayed by the first video source. 

2. The video display system of Claim 1 , wherein the second video source comprises a video 
driver and a video steerer, mounted with the video driver to direct optical energy therefrom 
to the display medium. 

3. The video display system of Claim 1 , wherein the first video source comprises a projector, 
the second video source comprises a projector and an image steerer mounted with the 
projector to direct optical energy therefrom to the display medium. 

4. The video display system of Claim 3, wherein the image steerer comprises a mirror 
capable of pan and tilt motion. 

5. The video display system of Claim 4, wherein the pan and tilt motion of the mirror is 
controlled by a computer. 

6. The video display system of Claim 4, wherein the first portion comprises the entire image 
and wherein the second portion is a subset of the entire image. 

7. The video display system of Claim 1 , wherein the image transformer comprises a 
homogeneous transform. 
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A method of displaying on a display mediunn an image, wherein the image comprises a 
first portion to be displayed at a first resolution and a second portion, mobile with respect 
to the first portion, to be displayed at a second resolution, comprising: 

a) Displaying with a first video source the first portion on the display medium; 

b) Configuring a second video source so that its output corresponds to the portion of the 
display medium corresponding to the second portion; 

c) Generating an input to the second video source, compensated for the relative 
configurations of the first video source, the second video source, and the display 
medium. 

The method of Claim 8, wherein the input to the second video source is compensated 
according to a transform determined by: 

a) Determining a first image plane transform by determining a correlation between an 
image plane corresponding to the first video source and the display medium; 

b) Determining a plurality of image plane correlations between the display on the display 
medium of the image plane corresponding to the first video source and an image 
plane corresponding to the second video source, each correlation corresponding to a 
configuration of the second video source; 

c) Determining a second image transform as a function of the configuration of the 
second video source from the first image plane transform and the image plane 
correlations. 

The method of Claim 9, wherein: 

a) the plurality of image plane correlations has sufficient number that the number of 
unknown parameters in the second image transform is less than the number of 
equations resulting from the image plane correlations; and 

b) the second image transform is determined from a linear regression on the image 
plane correlations. 
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A method of determining an image transform for registration of first and second images to 
be displayed on a display medium, wherein the second image can be displayed at various 
locations relative to the first image by changing the configuration of a second video 
source, wherein the display of the first image has an associated first image plane and the 
display of the second image has an associated second Image plane, comprising: 

a) Determining a first image plane transform by determining a correlation between the 
first image plane and the display medium; 

b) Determining a plurality of image plane correlations between the display on the display 
medium of the first image plane and the second image plane, each correlation 
corresponding to a configuration of the second video source; 

c) Determining the image transform as a function of the configuration of the second 
video source from the first image plane transform and the image plane correlations. 

The method of Claim 1 1 , wherein: 

a) the plurality of image plane correlations has sufficient number that the number of 
unknown parameters in the image transform is less than the number of equations 
resulting from the image plane correlations; and 

b) the image transform is determined from a linear regression on the image plane 
correlations. 
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Abstract of the Disclosure 

The present invention provides a method and apparatus for displaying two portions of an 
image at two resolutions- For example, the invention can display an entire image at a first 
resolution, and a subset of the image at a second, higher resolution. Two inexpensive, low 
5 resolution displays can be used to produce a large image with high resolution only where 
needed. 
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Figure 2 
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COMBINED DECLARATION AND POWER OF ATTORNEY 

As the below-named inventor(s): Barry L. Spletzer, George S. Davidson, Daniel J. 
Zimmerer, and Lisa C- Marron 

My residence, post office address, and citizenship are as stated below next to my name. 

I/We believe I/we am/are the original, first, and sole inventor (if only one name is 
listed below) or an original, first, and joint inventor (if plural names are listed below) of the 
subject matter which is claimed and for which a patent is sought on the invention entitled 
FRACTIONAL SCREEN VIDEO ENHANCEMENT APPARATUS, the 
specifications of which (check one) 

X is attached hereto 

was filed on as Serial No. and was amended on 

(if applicable). 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claims as amended by any amendment referred to above. 

I acknowledge the duty to disclose information which is material to the patentability of 
this application in accordance with Title 37, Code of Federal Regulations, 
§ 1.56(a). 

I hereby claim foreign priority benefits under Title 35, United States Code 
§ 119 of any foreign application(s) for patent or inventor's certificate listed below and have 
also identified below any foreign application for patent or inventor's certificate having a filing 
date before that of the application on which priority is claimed: 

PRIOR FOREIGN APPLICATION 

Priority Claimed 

|YES| I 
— INQi 12L. 

Number Country Filed (Day/Month/Year) 

I/We hereby claim the benefit under Title 35, United States Code, § 120 of any United 
States application(s) listed below and insofar as the subject matter of each of the claims of this 
application is not disclosed in the prior United States application in the manner provided by 
the first paragraph of Title 35, United States Code, § 112, I acknowledge the duty to disclose 
material information as defined in Title 37, Code of Federal Regulations, § 1.56(a) which 
occurred between the filing date of the prior application and the national or PCT international 
filing date of this application: 

NONE 
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POWER OF ATTORNEY: As the named inventors, 


we hereby appoint the following attorney(s) 


and/or agent(s) to prosecute this application and transact all business in the Patent and 
Trademark Office connected therewith. 


Names and Registration Nos. 


Names and Registration Nos. 


Timothy D. Stanley 30,428 
George H. Libman 27,984 | 
Kurt C. Olsen 29,961 , 1 
Russell D. Elliott 35,497 | 
V. Gerald Grafe 42,599 1 


Gregory A. Cone 28,835 
Brian W. Dodson 39,170 
John P. Hohimer 39,226 
Timothy P, Evans 41,013 
Elmer A. Klavetter 42,743 
Robert D. Watson P45,604 




Send Correspondence to: 


Direct Telephone Calls To: 


SANDIA NATIONAL LABORATORIES | 
Patent & Licensing Center | 
P. O. Box 5800 - MS-0161 | 
Albuquerque, New Mexico 87185-0161 | 


V. Gerald Grafe 
(505) 284-4404 



(I)/We hereby declare that all statements made herein of my/our own knowledge are true 
and that all statements made on information and belief are believed to be true; and further that 
these statements were made with the knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, under § 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issuing thereon. 
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Citizenship: USA 
DATE: 
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Inventor Name: Daniel J. Zimmerer Citizenship: USA 

SIGNATURE: Q / J 7 DATE: 

Q^^dfy^ ^^-t¥-07^ 

Residence (City & St^l^or Country): Tijeras, New Mexico 87059 
Postal Address: 42 Campo Rd., Tijeras, New Mexico 87059 



Inventor Name: Lisa C. Marron Citizenship: USA 

SIGNATURE: " <W DATE: 
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Residence (City & State or Country): Albuquerque, New Mexico 87123-1124 
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